Parser event ir - #1768
Draft
tompng wants to merge 3 commits into
Draft
Conversation
The Prism visitor now emits an intermediate representation: a flat stream of plain-data records (scope open/enter/exit events, methods, constants, comments, directives, ...) instead of mutating the store directly. A new CodeObjectBuilder replays the records in emission order and contains all CodeObject creation and name resolution logic, unchanged. Lexical state (module nesting syntax, visibility cursor, comment consumption, token extraction) stays on the emitter side; everything that reads or writes the store moves to the builder. Scopes are identified structurally by ids so that the builder reproduces the exact open-time container identity of the previous single-pass implementation, including scopes that resolve to no documentable container, whose inner records are dropped. This is a behavior-preserving refactoring step toward two-phase name resolution: a later change can collect the IR of all files first and resolve names against the complete declaration table instead of the parse-order store state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Batch documentation runs now parse every file into IR first and replay the IR of Ruby files afterwards, in the original file order. This is a step toward resolving names against the declarations of all files instead of the store state at parse time. Output for Ruby-only code bases is unchanged: the replay performs the same store mutations in the same order, only later in time. In a code base mixing C and Ruby sources, mutations of the C parser now happen before all Ruby mutations instead of interleaved in file order. A single #scan call still parses and builds immediately; the server's per-file reparse path keeps that behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A new NamespaceResolver runs before CodeObjects are built: it simulates the lexical scopes of the IR of every Ruby file in the batch and computes the declaration table (namespaces, constants and module aliases) with a fixed-point iteration, because resolving one declaration's owner can depend on the names another declaration introduces. Each IR record is annotated with its resolved full names: the declared class or module, the superclass, def receivers, constant owners, include/extend targets and constant alias targets. CodeObjectBuilder consumes the annotations and contains no name resolution of its own. Namespaces that no file has built yet are created ahead of the build as ignored ghosts; a ghost revives through the same mechanism as a namespace created inside a :stopdoc: region when some file contributes documentable contents, and stays out of the documentation otherwise. Resolution is a pure function of the declaration table, with two visible consequences. Names resolve against the declarations of all files regardless of the file order, so a superclass or mixin declared in a file built later is found. Within one file, forward declarations are visible as well; test expectations of position-dependent resolution are updated. Details ported from the single-pass behavior: a module named as a superclass is upgraded to a class (including for the implicit Object superclass), `class Cipher < Cipher` resolves the right-hand side to an outer namespace rather than the class the clause is defining, and the resolver ignores documentation suppression when collecting declarations: those declarations still define real Ruby constants, so they participate in name resolution. An implicit namespace - the owner of `class B::C` when no B is declared anywhere - is never invented while a real declaration could resolve the name: in real Ruby such code raises NameError unless something defined B first, so preferring a real declaration matches every load order that works. Nested undeclared roots stay pending until the table is stable and are then pinned as implicit namespaces, outermost first, so that one invented namespace can serve the deeper pendings. The kind of a namespace is tracked as class, module or unknown; an unknown kind falls back to a module when the namespace is created. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tompng
requested a deployment
to
fork-preview-protection
August 8, 2026 19:19 — with
GitHub Actions
Waiting
tompng
marked this pull request as draft
August 8, 2026 19:19
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request restructures the Ruby parser to emit an intermediate representation (IR), defers Ruby CodeObject construction until after all files in a batch are parsed, and preloads namespaces across the batch so constant/class/module name resolution becomes independent of file processing order.
Changes:
- Add IR-based Ruby parsing (
parse_ir+build_ir) and defer Ruby builds inRDoc::RDoc#parse_filesto enable cross-file namespace preloading before building. - Introduce
RDoc::Parser::Ruby::NamespaceResolver(batch fixed-point solver) andRDoc::Parser::Ruby::CodeObjectBuilder(replays IR to build CodeObjects). - Expand/adjust tests to cover order-independent resolution and additional parse_files edge cases.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/rdoc/rdoc_rdoc_test.rb | Adds parse_files regression tests for order-independent resolution and other multi-file edge cases. |
| test/rdoc/parser/ruby_test.rb | Updates expectations/sorting and clarifies assertions for new resolution behavior. |
| lib/rdoc/rdoc.rb | Defers Ruby parsing/building and adds batch build step with namespace preloading and improved parse error hint factoring. |
| lib/rdoc/parser/ruby.rb | Splits Ruby parsing into IR emission + build replay; emits IR records instead of mutating CodeObjects during AST walk. |
| lib/rdoc/parser/ruby_namespace_resolver.rb | New batch namespace resolution pass to create “ghost” namespaces before building. |
| lib/rdoc/parser/ruby_code_object_builder.rb | New IR replay engine that materializes comments/directives and constructs CodeObjects. |
| lib/rdoc/parser.rb | Ensures the new Ruby builder/resolver files are required when loading the parser subsystem. |
| lib/rdoc/code_object/class_module.rb | Adds namespace_ghost flag to support pre-created namespaces. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+267
to
272
| def build_ir(resolve: true) | ||
| NamespaceResolver.new(@store).preload_namespaces([self]) if resolve | ||
| builder = CodeObjectBuilder.new(@top_level, @store, @options, @stats, @preprocess, track_visibility: @track_visibility) | ||
| builder.run(@ir) | ||
| @top_level | ||
| end |
Comment on lines
+116
to
+123
| if payload[:startdoc] && !container.ignored? | ||
| # Compatibility: `module Net #:nodoc:` followed by :stopdoc:/:startdoc: | ||
| # regions is a common pattern that expects :startdoc: to make the | ||
| # container documentable again. Containers ignored here were created | ||
| # in a suppressed region and need documentable contents to revive. | ||
| container.start_doc | ||
| container.force_documentation = true | ||
| end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, RDoc's module/class resolve is parse-order dependant, and may fail.
Parse steps
Future possibilities
Server mode can be more stable